Automated pop-up display altering a user of a machine determined medical best practice recommendation

ABSTRACT

Provided are a computer program product, system, and method for determining real-time changes to content entered into a user interface to generate results for the content. In response to determining that entry of first content in a user input field rendered in a user interface is completed, the first content is provided to a classification program to classify into a first machine classification to provide to a rules engine to determine a first machine determined proposition. The first machine determined proposition is rendered in the user interface. A determination is made of second content in the user input field from the user that differs from the first content. The second content is provided to the classification program to classify into a second machine classification to provide to the rules engine to determine a second machine determined proposition rendered in the user interface with the second content.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a computer program product, system, andmethod for a user interface for determining real-time changes to contententered into the user interface to provide to a classifier program andrules engine to generate results for the content

2. Description of the Related Art

Natural Language Processing (NLP) algorithms are used to classifyinformation provided by a user via a computer user interface. In themedical field, NLP algorithms are used to process medical informationfrom patient records or from clinician free form narration (audio ortext) to extract clinical facts from the processed content. Theseextracted clinical facts may then be used to determine the completenessand accuracy of the gathered information and to alert the clinician ofconditions to further explore and develop, to provide alerts, and todetermine alternative hypothesis of diagnosis for the extracted facts.In this way, patient health records and doctor entered free formnarration may be processed to provide diagnosis of patient conditions.

There is a need in the art for improved techniques for extractinginformation from a user interface in which user observations andfindings are entered to process to determine results, actions and bestpractices for the extracted information.

SUMMARY

An embodiment may comprise a computer program product, system, andmethod for determining real-time changes to content entered into a userinterface to provide to a classifier program and rules engine togenerate results for the content. First content from a user in a userinput field is rendered in a user interface. A determination is madethat entry of the first content is completed. In response to determiningthat the entry of the first content is completed, the first content isprovided to a classification program to classify the first content intoa first machine classification to provide to a rules engine to apply aseries of rules to determine a first machine determined proposition forthe first machine classification. The first machine determinedproposition is rendered in the user interface. A determination is madeof second content in the user input field from the user that differsfrom the first content for which the first machine determinedproposition is rendered. The second content is provided to theclassification program to classify the first content into a secondmachine classification to provide to the rules engine to apply theseries of rules to determine a second machine determined proposition forthe second machine classification. The second machine determinedproposition is rendered in the user interface with the second content.

In a further embodiment, a first fingerprint is generated from the firstcontent that uniquely identifies the first content. A second fingerprintfrom the second content that uniquely identifies the second content. Adetermination is made as to whether the first fingerprint differs fromthe second fingerprint, wherein the second content is provided to theclassifier program in response to determining that the first fingerprintis different from the second fingerprint.

In a further embodiment, the user input field comprises a first userinput field. Determining that the first content is completed comprisesdetecting user input into a second input field in the user interface.

In a further embodiment, the first and the second content comprise userentered findings. The classifier program classifies the user enteredfindings as a machine classification. The first and second machinedetermined propositions each comprise at least one of referencematerial, calculations, summary of information from other sources, bestpractices, related to the machine classification.

In a further embodiment, the user input field comprises a first userinput field. The first and the second content comprises user enteredobservations. User input is received indicating a user classification ofthe user entered observations in a second user input field. Theclassifier program additionally receives the user classification in thesecond user input field with the user entered observations in the firstuser input field to process to determine the first and the secondmachine classifications to provide to the rules engine.

In a further embodiment, the user entered observations comprise medicalobservations of a patient, the user classification comprises a userdiagnosis of the user entered observations, the first and the secondmachine classifications comprise clinical diagnoses based on at leastone of the user entered medical observations and the user diagnosis, andthe first and second machine determined propositions from the rulesengine comprise first and second medical best practices based onapplying a series of rules of the rules engine to the clinical diagnosisfrom the classifier program. The term “best practices” may refer to bestpractice recommendations, best recommendations, and other types ofpreferred or relevant recommendations or actions, etc., to take based onan observed condition or diagnosis.

In a further embodiment, an action user interface is rendered with thesecond machine determined proposition and an acceptance control to allowa user to accept or reject the second machine determined proposition.The second machine determined proposition is rendered in the userinterface with the second content in response to receiving, via theacceptance control in the action user interface, indication that theuser accepted the second machine determined proposition.

In a further embodiment, the second machine determined proposition isrendered in the user interface with the second content and indicationthat the second machine proposition was not accepted in response toreceiving rejection of the rendered second machine determinedproposition through the action user interface.

In a further embodiment, at least one of a preferred classification anda preferred proposition is received from the user rejecting the secondmachine determined proposition with the acceptance control. A message issent with the at least one of the preferred classification and thepreferred proposition to an interface program managing access to theclassifier program and the rules engine. Including the preferredclassification in the rules engine trains the classification program toproduce the preferred classification based on the second content used toproduce the second machine determined proposition. Including thepreferred proposition and the preferred classification programs therules engine to associate the preferred proposition with the preferredclassification. Including the preferred proposition with no preferredclassification programs the rules engine to associate the preferredproposition with the second machine classification used to determine thesecond machine determined proposition.

In a further embodiment, content entered into the user input field iswritten into a log file. Content from the user input field is saved inthe log file as previously entered content. The log file is periodicallyprocessed by determining whether the content from the user input fieldwritten to the log file matches the previously entered content. Thedetermining the second content differs from the first content comprisesdetermining that the content from the user input field in the log filematches the previously entered content. Indication is made of thecontent from the user input field in the log file as the previouslyentered content. The content from the user input field in the log fileis sent to the classification program in response to the content in thelog file not matching the previously entered content to obtain a newresult from the rules engine to provide real time updating of the newresult in the user interface.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an embodiment of a computing environment.

FIG. 2a illustrates an embodiment of a user interface into which userfindings are entered and results based on the findings are rendered.

FIG. 2b illustrates an embodiment of a user interface in which machinegenerated best practices are rendered to a user.

FIG. 3 illustrates an embodiment of a log file.

FIG. 4 illustrates an embodiment of user content information used togenerate information in the user interface.

FIG. 5 illustrates an embodiment of operations to generate a reportthrough the user interface.

FIG. 6 illustrates an embodiment of operations to periodically process alog file to determine content to transmit to a classifier program.

FIG. 7 illustrates an embodiment of operations for a classifier programand rules engine to generate results from findings entered in the userinterface.

FIG. 8 illustrates an embodiment of operations to process results fromthe rules engine to render in the user interface.

FIG. 9 illustrates an embodiment of operations performed to retrain theclassifier program and/or rules engine based on a message rejecting amachine proposition.

FIG. 10 illustrates a computing environment in which the components ofFIG. 1 may be implemented

DETAILED DESCRIPTION

Described embodiments provide improvements to computer technology toprovide real-time machine propositions for user content, includingobservations and user classification of the observations, entered into auser interface via an input device, such as a keyboard or voice-to-textdevice, where the machine propositions may include actions,recommendations, reference material, calculations, summary ofinformation from other sources; best practices based on the findings,etc. For instance, a doctor or clinician may be entering observationsand possibly a diagnosis of a condition, such as by observing thepatient, lab test results, and medical images, and the clinician ordoctor may want to be immediately informed of the relevant bestpractices based on the observed findings. Further, the clinician ordoctor may want to see changes to the recommendations based on changesthe clinician or doctor makes to the findings entered into a medicalreport rendered in a user interface to provide real-time feedback onsuch changes. Further, other types of professionals wanting to obtainrecommended actions or best practices for observed findings, such asengineers analyzing a physical structure, device, manufacturing process,etc., or repair technicians may want to be informed real-time of theactions or best practices to take based on real-time entry of findingsand observations and modifications to the findings.

Described embodiments provide improvements to the computer technologyfor determining real-time entry or changes to inputted content, such asuser entered observations and classifications, entered into a userinterface to process and present to a program, such as a machinelearning classifier program and rules engine, to provide for real-timefeedback when making changes to the inputted observations. In describedembodiments, when first content, such as user inputted observations, isentered in the user interface, a determination is made when entry of thefirst content is completed. In response to entry of the first content,the first content is provided to a classification program to classifythe first content into a first machine classification to provide to arules engine to apply a series of rules to determine a first machinedetermined proposition for the first machine classification. The firstmachine determined proposition is rendered in the user interface. Adetermination is then made of second content in a user input field fromthe user, such as user entered observations, that differs from the firstcontent for which the first machine determined proposition is rendered.The second content is provided to the classification program to classifythe second content into a second machine classification to provide tothe rules engine to apply the series of rules to determine a secondmachine determined proposition for the second machine classification.The second machine determined proposition is rendered in the userinterface with the second content.

Further embodiments provide improvements to computer technology todetermine changes to the user entered content, such as observations andclassifications, entered into the user interface to provide to theclassifier program to provide real time-feedback on actions to takebased on the user entered findings. Content entered in to the user inputfield is written into a log file and content from the user input fieldin the log file is saved as previously entered content. To provide forreal time feedback and real-time machine determined propositions basedon changes to the user content entered into the user input field, thelog file is periodically processed to determine whether the content fromthe user input field written to the log file matches the previouslyentered content. The determining that current entered content differsfrom the previously entered and processed content comprises determiningthat the content from the user input field in the log file matches thepreviously entered content. Content from the user input field in the logfile is indicated as the previously entered content. The content fromthe user input field in the log file is sent to the classificationprogram in response to the content in the log file not matching thepreviously entered content to obtain a new result from the rules engineto provide real time updating of the new result in the user interface.

In certain embodiments, the user entered observations may apply toprocessing user entered medical observations of patient data,observation of the patient conditions and attributes, digital images,and physical samples, such as biopsies and bodily fluid samples, toprovide real-time feedback of best practices and recommendations to theuser entered medical observations. Upon determining changes in the userentered findings, such as observations and classifications, the newinputted observations may be sent to a classifier program to determinefrom the user entered medical observations a predefined machineclassification, such as a clinical diagnosis or recognized condition, toprovide to a rules engine. The rules engine determines medical bestpractices based on applying a series of rules from the rules engine tothe machine classification. The best practices and/or referencematerial, recommendations, calculations, summary of information fromother sources, actions and other relevant information may then beimmediately returned to render in the user interface to provideimmediate feedback, such as for best practices and changes to bestpractices for user entered observations.

FIG. 1 illustrates an embodiment of a computing environment in whichembodiments are implemented. A computing device 100 includes a processor102, a main memory 104, and a storage 106. The main memory 104 includesvarious program components including an operating system 108, a reportuser interface 200, a report generator 110 to generate the report userinterface 200, and input drivers 112 to interact with componentsconnected on a bus 114. The memory 104 further includes a log file 300in which content entered in the report user interface 200 is written,user content information 400 having information on content a user hasentered in the report user interface 200, and a report 116 that may begenerated from the content entered in the report user interface 200. Thereport user interface 200 may be continually displayed waiting to takeaction in response to user input.

The bus interface 114 may connect the processor 102, main memory 104, acommunication transceiver 118 to communicate (via wireless communicationor a wired connection) with external devices, including a network, suchas the Internet, a cellular network, etc.; a microphone 120 to transmitand receive sound for the personal computing device 100; a displayscreen 122 to render display output to a user of the personal computingdevice 100; a speaker 124 to generate sound output to the user; andinput controls 126 such as buttons and other software or mechanicalbuttons, including a keyboard, to receive user input. The componentsconnected on the bus 114 may communicate over one or more bus interfaces114 and other electronic interfaces.

The computing device 100 may communicate with a server 128 over anetwork 130 to transmit user content or findings and observationsentered into the report user interface 200 to a classifier program 132in the server 128 that processes the received user content to generate aclassification based on the user entered observation. For instance, ifthe user observations comprise observations and findings with respect toa medical examination, medical images (e.g., X-Ray or magnetic resonanceimaging digital image (MRI)), the classifier program 132 may producemachine classification, such as a diagnosis or description, of the userentered findings. In this way, the classifier program 132 presentsrelevant information based on user entered descriptions, such as aspecified pathology. The classifier program 132 may implement a machinelearning technique such as decision tree learning, association rulelearning, neural network, inductive programming logic, support vectormachines, Bayesian network, etc., to determine a classification based oncontent entered by the user in the report user interface 200, such asmedical observations and findings.

In one embodiment, the classifier program 132 may comprise a machinelearning program that is trained using a training set comprisingpreviously generated reports that have been classified with a groundtruth classification, and the classifier program 132 is trained toproduce the ground truth classifications provided for the training setof reports. For instance, if the training set comprises doctor enteredfindings, such as findings based on an Mill reading, then the providedground truths would be doctor determined classifications or clinicaldiagnosis based on those findings. The classifier program 132 would thenbe trained with those findings to produce the clinical diagnosisassigned to those findings and observations. The rules engine 134 wouldthen take that machine classification, such as a clinical diagnosis, anddetermine a proposition, such as a result, action or best practicesbased on the classification using a decision tree or table thatassociates specific courses of action or results with the classificationoutputted from the classifier program 132. For instance, if the outputof the classifier program 132 comprises a clinical diagnosis, then therules engine 134 would determine the best practices to treat a clinicaldiagnosis outputted from the classifier program 132, such as a drugtherapy, surgical treatment, further testing, further follow-up visit,etc. In this way, the rules engine 134 may provide a proposition, suchas an action or best practices, reference material, calculations,summary of information from other systems, etc., for each possibleclassified diagnosis output from the classifier program 132.

In one embodiment, the findings may comprise a size of an observedcondition on a patient, such as a size of an abdominal aortic aneurysms(AAA), observed features, such as size, shape, etc., of an incidentalthyroid nodule, ovarian cyst, non-incidental thyroid nodule, enlargedthyroid, simple ovarian cyst, etc. The rules engine 134 may specifyparticular best practices given different sizes of the AAA, such asrecommended follow-ups after so many years or a follow-up and anadditional vascular consultation. In certain embodiments, the content isprovided to the classifier program 132 in response to the radiologistadding text to the impressions section 206.

The server 128 may also include a rules engine 134 providing a decisiontree of rules to map received predefined machine classificationsoutputted from the classifier program 132 to produce a machinedetermined proposition based on the outputted machine classification,such as a result or action to take based on the classification. Forinstance, if the machine classification from the classifier program 132comprises a clinical diagnosis recognized by the rules engine 134, thenthe rules engine 134 may map that recognized clinical diagnosis to abest practices course of action to address that clinical diagnosis, suchdrug therapy, surgery, further testing, follow-up visit, etc. The server128 may further include a client interface 136 to interface and managecommunications with client systems to receive application calls from thereport generator 110 operating in multiple client computing devices 100,such as at different facilities, companies or places of business. Inalternative embodiments, the client interface 136 may receiveapplication calls from another component. An advantage of having therules engine 134 separate from the classifier program 132 is that therules engine 134 may be independently updated to provide new results,actions or best practices for the classified condition or diagnosisoutputted from the classifier program 132.

The network 130 may comprise one or more networks including Local AreaNetworks (LAN), Storage Area Networks (SAN), Wide Area Network (WAN),peer-to-peer network, wireless network, the Internet, etc.

The computing device 100 may store program components and correspondingdata, such as 108, 110, 112, 116, 200, 300 in a non-volatile storage106, which may comprise one or more storage devices known in the art,such as a solid state storage device (SSD) comprised of solid stateelectronics, NAND storage cells, EEPROM (Electrically ErasableProgrammable Read-Only Memory), flash memory, flash disk, Random AccessMemory (RAM) drive, storage-class memory (SCM), Phase Change Memory(PCM), resistive random access memory (RRAM), spin transfer torquememory (STM-RAM), conductive bridging RAM (CBRAM), magnetic hard diskdrive, optical disk, tape, etc. The storage devices may further beconfigured into an array of devices, such as Just a Bunch of Disks(JBOD), Direct Access Storage Device (DASD), Redundant Array ofIndependent Disks (RAID) array, virtualization device, etc. Further, thestorage devices may comprise heterogeneous storage devices fromdifferent vendors or from the same vendor.

The memory 104 may comprise a suitable volatile or non-volatile memorydevices, including those described above.

Generally, program modules, such as the program components 108, 110,112, 200, 132, 134, 136 may comprise routines, programs, objects,components, logic, data structures, and so on that perform particulartasks or implement particular abstract data types. The programcomponents and hardware devices of the computing device 100 and server128 of FIG. 1 may be implemented in one or more computer systems, whereif they are implemented in multiple computer systems, then the computersystems may communicate over a network.

The program components 108, 110, 112, 200 may be accessed by theprocessor 102 from the memory 104 to execute. Alternatively, some or allof the program components 108, 110, 112, 200, 132, 134, 136 may beimplemented in separate hardware devices, such as Application SpecificIntegrated Circuit (ASIC) hardware devices.

The functions described as performed by the programs 108, 110, 112, 200,132, 134, 136 may be implemented as program code in fewer programmodules than shown or implemented as program code throughout a greaternumber of program modules than shown.

FIG. 1 shows the classifier program 132 and rules engine 134 implementedin a server 128 over a network 130. In an alternative embodiment, theclassifier program 132 and rules engine 134 may be implemented in thecomputing device 100 including the report generator 110 and userinterface 200. In further embodiments, the server 128 may comprise acloud server providing cloud services for classifying user input withthe classifier program 132 and determining a course of action or bestpractices for the classification of the classifier program 132.

FIG. 2a illustrates an embodiment of a user interface 200 in which auser may input data, and includes a observations section 202 in which auser may enter observations about an entity being diagnosed, such as aperson, device, natural phenomena. The user may describe observationsfrom an image or video, direct observation of a phenomena, etc. The usermay further enter impressions 206, such as classifications andconclusions, in an impressions section 208 providing a classification ofthe observations, such as a condition, medical diagnosis, naturalphenomena, state of a device, etc. The impressions section 208 mayfurther render an action, solution, best practices, etc. 210 produced bythe classifier program 132 and rules engine 134. The classifier program132 may receive the user entered observations 302 and optionally theuser classification 304, such as a diagnosis or conclusion ofobservations the user inters in field 206 of the user interface 200, andclassify into a machine classification, such as a clinical diagnosis,category, etc., which is then provided to the rules engine 134 todetermine a machine proposition, e.g., result, action, recommendation,best practices, etc. 210 to render in the user interface 200 concerningthe user machine classified findings.

In certain embodiments, the report user interface 200 may be continuallydisplayed waiting to take action in response to user input into theimpressions section 208 providing the classification.

FIG. 2b illustrates an embodiment of an alert user interface 220, whichmay be displayed as a dialog box or pop-up alert, rendered on top of theuser interface 200, that alerts the user of a machine determinedproposition 222 based on the user entered content and findings 204 and206, which may comprise a machine determined best practices from therules engine 134 based on the user entered observations andclassification, or diagnosis, of the observations. The user may selectan accept selection control 224 to cause the machine determined bestpractices 222 to be entered into field 210 of the user interface 200 toaccept the machine generated best practices from the rules engine 134.The user may also select a reject selection control 226 to reject themachine determined best practices. The user may enter the machinedetermined best practices 22 into the report user interface 200 by voicecommand, copy and paste from the field 222 in the alert user interface220 into the best practices field 210 of the user interface 200, orclick a button, such as 224 and 226 to cause the automatic insertion ofthe best practices recommendation in the field 222 of the alert userinterface 220 into the user interface 200, such as into field 210.

The user interface 200 provides improvements to computer technology forrendering machine generated results, such as from a machine learningclassifier program 132 and rules engine 134, by, in real-time, providinguser entered observations 204 to the classifier program 132 and rulesengine 134 to allow immediate computation of new machine propositionsbased on user entered findings, which include new observations 204 anduser entered classifications 206, to immediately display in real-timethe machine propositions 210, such as recommended actions, bestpractices, actions, calculations, reference material, summarizedinformation from other sources, etc., in the user interface 200 as theyare being entered in the observations 202 section. Further, at somepoint the observations 204, user entered classifications 206, andmachine propositions 210 may be saved as a report 116 for later use. Infurther embodiments, other types of content and findings, in addition toor different from observations and classifications, may be entered inthe user interface.

FIG. 3 illustrates an embodiment of a log file 300 to which contententered in the user interface 200 is written, including observations 302entered in the observations section 202 of the user interface 200, suchas exams and other patient related information; impressions 304comprising the user entered classifications 206 entered in theimpressions section 208; and the machine propositions 306 from the rulesengine 134 from processing the machine classification produced by theclassifier program 132, which is rendered as a machine determinedproposition 210, e.g., best practices, etc. 210 in the impressionssection 208.

FIG. 4 illustrates an embodiment of user content information 400generated by the report generator 110 to determine when to transmit theobservations 302 to the classifier program 132, and includes a previoususer content fingerprint 402 comprising a unique fingerprint/hash codecalculated from a previous state of the user entered impressions 304 anda current user content fingerprint 404 comprising a uniquefingerprint/hash code calculated from a current state of the userentered impressions 304. The fingerprints 402 and 404 may be calculatedusing a fingerprinting algorithm to generate a unique code based oninput, such as a fingerprinting method, or hash code generatingalgorithm.

FIG. 5 illustrates an embodiment of operations performed by the reportgenerator 110 to render the report user interface 200, rendered on thedisplay screen 122, to generate a report 116. Upon initiating (at block500) an operation to generate a report 116 as part of a start of areport generations session, the report generator 110 receives (at block502) user input via input controls 126, such as a keyboard mouse,touchscreen, or microphone 120 to renders in the report user interface200. The report generator 110 continually writes (at block 504) userinput entered into the observations 202 and impressions 208 sectionsinto observations 302 and impressions (e.g., classifications) 304fields, respectively, of the log file 300. In this way, the log file 300is updated in real time with new information the user enters into theobservations 202 and impressions 208 sections.

FIG. 6 illustrates an embodiment of operations performed by the reportgenerator 110 to periodically process the log file 300 to implementreal-time updating of the machine determined propositions 210 renderedin the user interface 200. The report generator 110 may periodicallyprocess the log file 300 after a time period, such as a fraction of asecond or in response to the writing to the log file 300, etc. Uponprocessing (at block 600) the log file 300, the report generator 110processes (at block 602) the content of the observations field 302 togenerate a content fingerprint/hash code comprising a unique identifierof the user entered findings. If (at block 604) this is the firstprocessing of the log file 300 for a report generation sessions, thenthe report generator 110 determines (at block 606) whether the userentered information in the impressions field 304, which may in certainembodiments signal that the user has completed entering the observations302. In further implementations, the report generator 110 maycontinuously provide feedback entered through the user interface 200without waiting for the user to input content into the impressionssection 208. Other tests may be used to determine that the entering ofthe observations 304 has completed, such as selection of a button oruser interface element. If (at block 606) a determination is made theuser completed entering observations 304, then the content fingerprintcalculated from the current observations 302 is stored (at block 608) asthe current user content fingerprint 404. The user entered findings,such as observations 302 and impressions 304 are transmitted (at block610) to the classifier program 132.

If (at block 604) this is not the first processing of the log fileduring a report generation session to generate a report, then thecurrent user content fingerprint 404 is indicated (at block 614) as theprevious user content fingerprint 402 and the generated contentfingerprint is stored (at block 614) as the new current user contentfingerprint 404. If (at block 616) the current 404 and previous 402 usercontent fingerprints do not match, then the user entered observations304 has changed since last checked and the report generator 110transmits the new user entered observations 302 to the classifierprogram 132 to generate new results. If (at block 616) the fingerprints402 and 404 are not different, then control ends as there are no newimpressions 304 to determine if there are new results, actions or bestpractices to render.

The embodiment of FIG. 6 provides improved computer technology toprovide an immediate and real-time determination if new userobservations 204 have been entered in the user interface 200 to providefor real-time updating of the results by immediately sending any changeduser entered impressions 304 to the classifier program 132 and rulesengine 134 to process. Any new results from the rules engine 134 may bedisplayed in the alert user interface 220 (FIG. 2B) in the determinedproposition field 222, which the user may then cause to be inputted intothe report 200. This allows for the immediate updating of the machinedetermined propositions 210 rendered in the alert user interface 220based on new user entered findings, such as observations and/orimpressions/classifications.

FIG. 7 illustrates an embodiment of operations performed by theclassifier program 132 and the rules engine 134 to process observations304 from the user to generate results. Upon receiving (at block 700)information from the user impressions 304 and other information, such asentered in observations 204, a classification of the observations 302,metadata, etc., the classifier program 132 classifies (at block 702) theentered information, e.g., impressions 304, etc., into a predefinedmachine classification, such as a clinical diagnosis or recognizedcondition. In additional embodiments, further information may beinputted into the classifier program 132, such as the user impressions206, or user classification, and additional information from othersources. For instance, for medical diagnostics, in addition to receivingthe user observations 204 and impressions 206, the classifier program132 may receive medical lab results, information from other reports orsources, etc., all that may assist the classifier program 132 to producea more accurate machine classification of the user entered findings,e.g., observations 302 and/or impressions 304. The classifier program132 sends (at block 704) the determined machine classification to therules engine 134.

Upon receiving the machine classification, the rules engine 134 applies(at block 706) a decision tree of rules to the machine classification todetermine a machine proposition, such as an action to take, bestpractices, etc., for the provided classification. The determined resultis returned (at block 708) to report generator 110 to render the resultin the alert user interface 220 in field 222 to allow the user to addthe determined result to the report. The report generator 110 or anotherprogram component may generate and manage the alert user interface 220.

With the embodiment of FIG. 7, user entered findings, such asobservations and/or impressions, entered in real time are sent to theclassifier program 132 and the rules engine 134 to provide immediatemachine propositions, e.g., actions, best practices, etc. to return tothe report generator 110 to immediately render the results in the alertuser interface 220, such as field 222. This enables to the user to seein real-time the changes in the results, actions or best practicesresulting from entering new findings, and determine whether to have thementered into the report being generated.

FIG. 8 illustrates an embodiment of operations performed by the reportgenerator 110 to process results received from the rules engine 134.Upon receiving (at block 800) a result, recommendation or action fromthe rules engine 134 in response to the sent observations 302, thereport generator 110 determines (at block 802) whether the receivedmachine proposition matches the machine proposition 306 saved in the logfile currently rendered in field 210 of the user interface 200. If (atblock 802) there is a match, then the machine proposition has notchanged due to the change in the findings, e.g., observations andimpressions, and control ends. If (at block 802) there is not a match,meaning the machine proposition or suggested action has changed sincethe previous findings, then the report generator 110 displays (at block804) a dialog box, such as the alert notification dialog box 220 in FIG.2b , showing the received machine determined recommendations 222 andgraphical controls 224, 226 to allow the user to accept or reject,respectively, the new result in the dialog box 220. If (at block 806)the user has indicated to accept the received machine proposition, byselect the accept graphical control 224, then the report generator 110renders (at block 808) the received machine proposition in the field 210in the user interface, e.g., in impressions field 208 and refresh theuser interface display 200 with the new machine proposition. In furtherembodiments, the user can also copy/paste or use a provided macro tomove the machine proposition anywhere into the report.

The new machine proposition is written (at block 810) to the machineproposition field 306 in the log file 300. If (at block 806) the userrejects the received machine proposition, such as by selecting thereject graphical control 226, then the received machine proposition isindicated in a field of the user interface 220, such as the action field210 that the user, and indication is also made that the user, such asradiologist, disagreed with the machine proposition, such as actions,recommendations, reference material, calculations, summary ofinformation from other sources; best practices based on the findings,etc. A user interface is then rendered (at block 814) to allow the userto specify a preferred proposition, e.g., best practices, and/or apreferred classification based on the current content in the userinterface 200, as well as additional user feedback or comments. Uponreceiving the user input, the report generator 110 sends (at block 816)a message to the client interface 136 at the server 128 with informationon the rejected machine proposition, preferred proposition/preferredclassification, the content (observations, impressions, etc.) that waspreviously sent to generate the rejected machine determined proposition,and any other feedback.

With the embodiment of FIG. 8, machine determined propositions areprovided real time in response to the recently sent findings. Inresponse, the report generator 110 renders an alert notification dialog200 to allow the user to review the results, such as machine determinedpropositions, actions to take, best practices for medical diagnosis,etc., and then determine whether to accept the machine proposition toinclude into the report or user interface 200 being generated. Further,described embodiments provide improved techniques for allowing the userto provide input on the rejected machine proposition to improve thecalculations and operations of the classifier program 132 and rulesengine 134 by providing a user preferred classification and/or preferredproposition based on the current content rendered in the user interface200 to the server 128 to retrain the rules engine 134 and/or classifierprogram 132 to further optimize operations and results. Further, therejected machine propositions may be entered into the log file 300.

FIG. 9 illustrates an embodiment of operations performed by the clientinterface 136, classifier program 132 and/or rules engine 134 to processa message sent from the report generator 110 where the user rejects theprovided machine proposition. Upon receiving (at block 900) the message,including information on the rejected machine proposition, preferredproposition/preferred classification, the content (observations,impressions, etc.) that was previously sent to generate the rejectedmachine proposition, and any other feedback, if (at block 902) apreferred classification was provided, then the machine learningalgorithm of the classifier program 132 is trained (at block 904) tooutput the preferred classification as the machine classification basedon the previously sent content that resulted in the rejected machineproposition. If (at block 906) a preferred proposition is provided, thenthe rules engine 134 is updated (at block 908) to output the preferredproposition as the machine proposition for the preferred classification,which the classifier program 132 has been retrained to produce for thegiven current content. The rules engine 134 may further be updated (atblock 910) to not associate the rejected machine proposition with themachine classification used to produce the rejected machine proposition,so as not to produce again the machine proposition that the userrejected given the current content, findings, impressions, observations,etc. in the user interface 200.

If (at block 902) a preferred classification was not provided and if (atblock 912) a preferred proposition was provided, then the rules engine134 is updated (at block 914) to output the preferred proposition as themachine proposition for the machine classifier used to produce therejected machine proposition, so that the user suggested preferredproposition will be produced instead of the user rejected machineproposition for that machine classifier. From the no branch of block912, block 914 or 910, any additional comments provided with the messageare forward to a server administrator to consider for improving theperformance of the classifier program 132 and/or rules engine 134.

The embodiment of FIG. 9 provides improved computer technology forretraining the classifier program 132 and/or rules engine 134 to produceuser preferred output when the user rejects a machine propositionpreviously sent to allow continual improvement of the machinepropositions produced for given content, such as improved actions, bestpractices, etc.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer program product comprises a computer readable storagemedium implemented using standard programming and/or engineeringtechniques to produce software, firmware, hardware, or any combinationthereof. The described operations may be implemented as code or logicmaintained in a “computer readable storage medium”. The term “code” and“program code” as used herein refers to software program code, hardwarelogic, firmware, microcode, etc. The computer readable storage medium,as that term is used herein, includes a tangible element, including atleast one of electronic circuitry, storage materials, a casing, ahousing, a coating, hardware, and other suitable materials. A computerreadable storage medium may comprise, but is not limited to, a magneticstorage medium (e.g., hard disk drives, floppy disks, tape, etc.),optical storage (CD-ROMs, DVDs, optical disks, etc.), volatile andnon-volatile memory devices (e.g., EEPROMs, ROMs, PROMs, RAMs, DRAMs,SRAMs, Flash Memory, firmware, programmable logic, etc.), Solid StateDevices (SSD), computer encoded and readable punch cards, etc. Thecomputer readable storage medium may further comprise a hardware deviceimplementing firmware, microcode, etc., such as in an integrated circuitchip, a programmable logic device, a Programmable Gate Array (PGA),field-programmable gate array (FPGA), Application Specific IntegratedCircuit (ASIC), etc. A computer readable storage medium is not comprisedsolely of transmission signals and includes physical and tangiblecomponents. Those skilled in the art will recognize that manymodifications may be made to this configuration without departing fromthe scope of the present invention, and that the article of manufacturemay comprise suitable information bearing medium known in the art.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The computational components of FIG. 1, including the computer device100 and the server 128, may be implemented in one or more computersystems, having a computer architecture as shown in FIG. 10, andincluding a processor 1002 (e.g., one or more microprocessors andcores), a memory 1004 (e.g., a volatile memory device), and storage 1006(e.g., a non-volatile storage, such as magnetic disk drives, solid statedevices (SSDs), optical disk drives, a tape drive, etc.). The storage1006 may comprise an internal storage device or an attached or networkaccessible storage. Programs, including an operating system 1008 andapplications 1010 stored in the storage 1006 are loaded into the memory1004 and executed by the processor 1002. The applications 1010 mayinclude the report user interface 200, report generator 110, inputdrivers 112, classifier program 132, rules engine 134, and clientinterface 136 and other program components described above. Thearchitecture 1000 further includes a network card 1012 to enablecommunication with the network 16. An input device 1014 is used toprovide user input to the processor 1002, and may include a keyboard,mouse, pen-stylus, microphone, touch sensitive display screen, or anyother activation or input mechanism known in the art. An output device1016, such as a display monitor, printer, storage, etc., is capable ofrendering information transmitted from a graphics card or othercomponent. The output device 1016 may render the GUIs described withrespect to figures and the input device 1014 may be used to interactwith the graphical controls and elements in the GUIs described above.The architecture 1000 may be implemented in any number of computingdevices, such as a server, mainframe, desktop computer, laptop computer,hand held computer, tablet computer, personal digital assistant (PDA),telephony device, cell phone, etc.

The terms “an embodiment”, “embodiment”, “embodiments”, “theembodiment”, “the embodiments”, “one or more embodiments”, “someembodiments”, and “one embodiment” mean “one or more (but not all)embodiments of the present invention(s)” unless expressly specifiedotherwise.

The terms “including”, “comprising”, “having” and variations thereofmean “including but not limited to”, unless expressly specifiedotherwise.

The enumerated listing of items does not imply that any or all of theitems are mutually exclusive, unless expressly specified otherwise.

The terms “a”, “an” and “the” mean “one or more”, unless expresslyspecified otherwise.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or moreintermediaries.

A description of an embodiment with several components in communicationwith each other does not imply that all such components are required. Onthe contrary a variety of optional components are described toillustrate the wide variety of possible embodiments of the presentinvention.

When a single device or article is described herein, it will be readilyapparent that more than one device/article (whether or not theycooperate) may be used in place of a single device/article. Similarly,where more than one device or article is described herein (whether ornot they cooperate), it will be readily apparent that a singledevice/article may be used in place of the more than one device orarticle or a different number of devices/articles may be used instead ofthe shown number of devices or programs. The functionality and/or thefeatures of a device may be alternatively embodied by one or more otherdevices which are not explicitly described as having suchfunctionality/features. Thus, other embodiments of the present inventionneed not include the device itself.

The foregoing description of various embodiments of the invention hasbeen presented for the purposes of illustration and description. It isnot intended to be exhaustive or to limit the invention to the preciseform disclosed. Many modifications and variations are possible in lightof the above teaching. It is intended that the scope of the invention belimited not by this detailed description, but rather by the claimsappended hereto. The above specification, examples and data provide acomplete description of the manufacture and use of the composition ofthe invention. Since many embodiments of the invention can be madewithout departing from the spirit and scope of the invention, theinvention resides in the claims herein after appended.

1-22. (canceled)
 23. A computer program product for alerting a user of amachine determined proposition comprising a medical best practicerecommendation, the computer program product comprising a computerreadable storage medium having computer readable program code thereonthat is executable to perform operations, the operations comprising:subsequent to a user entering a medical observation of a patient in auser interface, automatically rendering an alert indicating that amachine determined proposition comprising a medical best practicerecommendation has been determined that is associated with the medicalobservation entered by the user, wherein the alert is renderedsimultaneously with the medical observation of the patient; rendering anaccept selection control that is selectable by the user and associatedwith the machine determined medical best practice recommendation;receiving user selection of the accept selection control associated withthe machine determined medical best practice recommendation; andrendering the machine determined medical best practice recommendation inthe user interface simultaneously with the medical observation inresponse to receiving the user selection of the accept selectioncontrol.
 24. The computer program product of claim 23, whereinautomatically rendering an alert comprises automatically rendering apop-up alert.
 25. The computer program product of claim 23, wherein theoperations further comprise: rendering in the user interface, togetherwith the machine determined medical best practice recommendation and themedical observation, a medical reference citation associated with themachine determined medical best practice recommendation in response toreceiving the user selection of the accept selection control.
 26. Thecomputer program product of claim 23, wherein the operations furthercomprise: rendering a reject selection control that is selectable by theuser and associated with the machine determined medical best practicerecommendation; receiving user selection of the reject selection controlassociated with the machine determined medical best practicerecommendation; and rejecting the machine determined medical bestpractice recommendation in response to receiving the user selection ofthe reject selection control.
 27. Apparatus for alerting a user of amachine determined proposition comprising a medical best practicerecommendation, comprising: at least one processor; and at least onecomputer readable storage medium, wherein the at least one computerreadable storage medium comprises computer program instructionsexecutable by the at least one processor to perform operations, theoperations comprising: subsequent to a user entering a medicalobservation of a patient in a user interface, automatically rendering analert indicating that a machine determined proposition comprising amedical best practice recommendation has been determined that isassociated with the medical observation entered by the user, wherein thealert is rendered simultaneously with the medical observation of thepatient; rendering an accept selection control that is selectable by theuser and associated with the machine determined medical best practicerecommendation; receiving user selection of the accept selection controlassociated with the machine determined medical best practicerecommendation; and rendering the machine determined medical bestpractice recommendation in the user interface simultaneously with themedical observation in response to receiving the user selection of theaccept selection control.
 28. The apparatus of claim 27, whereinautomatically rendering an alert comprises automatically rendering apop-up alert.
 29. The apparatus of claim 27, wherein the operationsfurther comprise: rendering in the user interface, together with themachine determined medical best practice recommendation and the medicalobservation, a medical reference citation associated with the machinedetermined medical best practice recommendation in response to receivingthe user selection of the accept selection control.
 30. The apparatus ofclaim 27, wherein the operations further comprise: rendering a rejectselection control that is selectable by the user and associated with themachine determined medical best practice recommendation; receiving userselection of the reject selection control associated with the machinedetermined medical best practice recommendation; and rejecting themachine determined medical best practice recommendation in response toreceiving the user selection of the reject selection control.
 31. Acomputer implemented method for alerting a user of a machine determinedproposition comprising a medical best practice recommendation,comprising performing with one or more computers: subsequent to a userentering a medical observation of a patient in a user interface,automatically rendering an alert indicating that a machine determinedproposition comprising a medical best practice recommendation has beendetermined that is associated with the medical observation entered bythe user, wherein the alert is rendered simultaneously with the medicalobservation of the patient; rendering an accept selection control thatis selectable by the user and associated with the machine determinedmedical best practice recommendation; receiving user selection of theaccept selection control associated with the machine determined medicalbest practice recommendation; and rendering the machine determinedmedical best practice recommendation in the user interfacesimultaneously with the medical observation in response to receiving theuser selection of the accept selection control.
 32. The computerimplemented method of claim 31, wherein automatically rendering an alertcomprises automatically rendering a pop-up alert with the one or morecomputers.
 33. The computer implemented method of claim 31, wherein theoperations further comprise performing with the one or more computersrendering in the user interface, together with the machine determinedmedical best practice recommendation and the medical observation, amedical reference citation associated with the machine determinedmedical best practice recommendation in response to receiving the userselection of the accept selection control.
 34. The computer implementedmethod of claim 31, wherein the operations further comprise: rendering,with the one or more computers, a reject selection control that isselectable by the user and associated with the machine determinedmedical best practice recommendation; receiving, with the one or morecomputers, user selection of the reject selection control associatedwith the machine determined medical best practice recommendation; andrejecting, with the one or more computers, the machine determinedmedical best practice recommendation in response to receiving the userselection of the reject selection control.